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Early Start to Emancipation Preparation (ESTEP)-Tutoring 
Executive Summary 



Approximately 510,000 children lived in out-of-home care on September 30, 2006, the most 
recent date for which national estimates are available. In fiscal year 2006, over 26,000 youths 
remained in care until they were legally “emancipated” to “independent living,” usually due to 
reaching age 18 or upon graduation from high school. On average, these youths have limited 
education and employment experience, relatively poor mental and physical health, and a 
relatively high likelihood of experiencing unwanted outcomes such as homelessness, 
incarceration, and nonmarital pregnancy. 

The Foster Care Independence Act of 1999 amended Title IV-E to create the John Chafee Foster 
Care Independence Program (CFCIP), giving states more funding and greater flexibility in 
providing support to youth making the transition to independent living. It also required 
evaluation of such services. The U.S. Department of Health and Human Services’ Children’s 
Bureau contracted with the Urban Institute and its partners — the Chapin Hall Center for Children 
and the National Opinion Research Center at the University of Chicago — to conduct this 
evaluation. The goal of this study is to determine the impacts of the programs funded under 
CFCIP in achieving key outcomes for youth. Four programs are being evaluated under this 
contract, and this report provides results for one of those programs. 



Description of ESTEP-Tutoring 

The subject of this report is the Early Start to Emancipation Preparation (ESTEP) -Tutoring 
program of Los Angeles County, California. ESTEP-Tutoring offers a service (i.e., tutoring) that 
is provided in numerous locations throughout the United States. While ESTEP-Tutoring may 
present a typical set of services, there are unusual aspects of the program that may provide useful 
information for other independent living programs. The program offers services beyond tutoring, 
including a mentoring relationship with the tutor and access to other independent living 
workshops provided through the ESTEP program. Aside from programmatic aspects, the 
program was selected because of the large number of youths that it serves (between 400 and 500 
annually at the time of the evaluation). At the time of the evaluation, ESTEP-Tutoring was an 
oversubscribed service, having nearly twice as many youths referred as program participants. 

The ESTEP-Tutoring program was created in 1998 to improve reading and math skills of foster 
youths age 14 and 15 who are one to three years behind grade level in reading or math. The 
program is also designed to empower youths to use other educational services and resources that 
may be available to them. ESTEP-Tutoring is offered through 12 community colleges in Los 
Angeles County by college student tutors. Tutoring is provided primarily in the home of the 
youth being served. It is based on an individual learning model in which tutors are trained to 
assess students’ skills in math, reading, and spelling. Curriculum materials are supplied at that 
skill level so that students may leam at their own pace. The tutoring relationship fosters a 
mentoring relationship between the tutor and the youth. Each youth is eligible for 65 hours of 
tutoring (this includes 15 hours tutors can use for preparation, mentoring, and other activities). 
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The Evaluation 



The evaluation consists of two components: an impact study involving three in-person interviews 
over two years and a process study. The sampling frame for the evaluation was all foster youths 
referred for ESTEP-Tutoring during the study period. Referred youths were assessed by the 
program. Those deemed appropriate for tutoring were randomly assigned to either the treatment 
group, referred to as “ESTEP group,” or the control group. The analytical sample consists of 445 
youth who were referred to the ESTEP-Tutoring program. The response rate exceeded 95 
percent, with over 90 percent of the baseline sample interviewed at the end of two years. Overall, 
61.8 percent of all ESTEP youth started tutoring and on average, youth received 18 hours of 
math tutoring and 17 hours of reading (language) tutoring. 

The conditions for referral to ESTEP-Tutoring are not always met according to our process 
study. An examination of referrals shows that a significant portion of youths did not meet the 
criterion based on the assessments at the time of the referral. A rough estimate based on the 
youths referred to the study indicates that nearly 14 percent of all youths had both reading and 
math grade-level equivalents greater than three years behind their grade level. Conversely, more 
than 2 percent were at or above their grade level for both measures. 

As in other field experiments involving social services where the evaluation staff s control over 
program receipt is not complete, some members of the control group received tutoring. In total, 
27 control youth (12.3 percent of the control group) received the service. They received 19 hours 
of reading and 19 hours of math on average, similar to the dosage of the ESTEP group. It is not 
clear from the process study how the control group youths accessed the service. 



Impact Findings 

Outcomes evaluated align closely with the program’s primary goals of improving reading and 
math skills and empowering youths to use other educational resources. At each of the three 
interview waves, youths completed three tests from the Woodcock Johnson Tests of 
Achievement III (a standard set of measures used in the field): letter- word identification, 
calculation, and passage comprehension. The unit of measurement was the age percentile , which 
indicates youths’ percentile rankings based on a normative sample. In addition, youths were 
asked about what grades they had received during their last full semester of school attendance, 
about their highest grade completed, and about their school behavior. Covariates in the analysis 
included physical and mental health, substance abuse, level of social support, and deviant 
behavior. 

The program had no impacts on educational outcomes. No statistically significant differences 
were observed between the ESTEP and control groups in any of the outcomes at the second 
follow-up. 

For both the ESTEP and the control groups, there were significant decreases in the age percentile 
averages between baseline and the second follow-up interview in the Woodcock Johnson letter- 
word identification and calculation tests. This indicates that the sampled youths lost ground 
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during this time vis-a-vis their normative age cohort on these measures. There was, however, a 
significant increase from baseline to the second follow-up in the average percentile score for 
passage comprehension. There were no significant changes over time in grades for the sample as 
a whole. Youths reported moderate to low levels of school-related problems. 

Youths assigned to the ESTEP group were more likely to have received educational tutoring at 
home than control group youths. Yet youths who did not participate in ESTEP-Tutoring but 
reported some exposure to other forms of tutoring were more likely to have received school- 
based tutoring. 



Lessons for Independent Living Programs 

The evaluation provides ample evidence of the continuing prevalence of education deficits 
among foster youth. Given these deficits, access to tutoring would seem to be a reasonable 
service to provide foster youth to help prepare them for independent living. However, with 
respect to the ESTEP-Tutoring program, our impact evaluation did not find compelling evidence 
that this program had any beneficial impact on the outcomes we assessed. 

If the situation in Los Angeles County is typical of other urban areas in the United States, our 
results suggest that tutoring is now fairly readily available to foster youth. While the process 
study provided evidence of the need for more tutoring resources for foster youths in Los 
Angeles, our impact findings indicate that most foster youths identified as being in need of extra 
help do get some form of tutoring. This has significant implications for the evaluation of tutoring 
programs targeting foster youths, making it difficult to establish a control group that does not 
have access to some other tutoring program. Further evaluation of which educational supports 
work for foster youths is necessary. 

ESTEP-Tutoring is based on the assumption that identifying skill deficits and addressing those 
deficits through tutoring, regardless of the educational program in a youth’s particular school, 
will help a youth succeed in school. Tutors may find it difficult to engage youths in the tutoring 
process when the youths see little relationship between tutoring activities and the educational 
material being covered at school. Likewise, tutoring may not have much of an impact on 
educational performance, particularly in the short term, if it does not directly address the skills 
being taught at school. 

Relatively few tutoring relationships evolved into longer-term mentoring relationships. This calls 
into question the notion that tutoring, at least tutoring of limited duration such as that offered by 
ESTEP during this evaluation, is very likely to leave foster youths with long-term adult 
relationships. 

The overall educational trajectories of the ESTEP and control groups over the two-year follow- 
up period indicate that not enough is done to address the educational needs of foster youths 
deemed at risk of educational failure. While ESTEP-Tutoring did not have significant impacts on 
the key outcomes assessed, additional models should be developed and assessed to help foster 
youths who are struggling in school. 
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Introduction 



Approximately 510,000 children lived in out-of-home care as of September 30, 2006, the most 
recent date for which national estimates are available (U.S. Department of Health and Human 
Services [DHHS] 2008). This represents about six children and adolescents per thousand. The 
vast majority of children and youth will exit care to what are considered permanent placements. 
Of the estimated 289,000 children who left out-of-home care in the United States during fiscal 
year 2006, 86 percent went to live with family, were adopted, or were placed in the home of a 
legal guardian (DHHS 2008). A few (2 percent) were transferred to another public agency, such 
as a probation or mental health department, and a few (2 percent) ran away and were discharged 
from care. Nine percent, or 26,517, remained in care until they were legally “emancipated” to 
“independent living,” usually due to reaching the age of majority or upon graduation from high 
school. In practice, few states allow youth to remain in care much past their 18th birthday 
(Bussey et al. 2000). About 5 percent (21,834) of all children and youths living in out-of-home 
care were between 18 and 21 years old. 

Research findings suggest that the transition to adulthood for foster youth in the United States is 
difficult. On average, they have had poor educational experiences, leading them to bring to the 
transition very limited human capital upon which to build a career or economic assets. They also 
often suffer from mental health problems that can negatively affect other outcome domains, and 
these problems are less likely to be treated once youth leave care. In addition, foster youth 
frequently become involved in crime and with the justice and corrections systems after aging out 
of foster care. Further, their employment prospects are bleak, and few of them escape poverty 
during the transition. At the same time, many former foster youth experience homelessness and 
housing instability after leaving care. Compared with their peers, former foster youth have higher 
rates of out-of-wedlock parenting. Interestingly, in spite of court-ordered separation from their 
families, often for many years, most former foster youth rely on their families to some extent 
during the transition to adulthood, though this is not always without risk (Barth 1990; Bussey et 
al. 2000; Cook, Fleischman, and Grimes 1991; Courtney et al. 2001; Courtney et al. 2005; 
Dworsky and Courtney 2000; Fanshel, Finch, and Grundy 1990; Festinger 1983; Frost and Jurich 
1983; Goerge et al. 2002; Harari 1980; Jones and Moses 1984; Mangine et al. 1990; Pecora et al. 
2005; Pettiford 1981; Sosin, Coulson, and Grossman 1988; Sosin, Piliavin, and Westerfeldt 
1990; Susser et al. 1991; Zimmerman 1982). 

These poor outcomes suggest the need for services to better prepare foster youth for the 
transition to adulthood. Two decades ago, there were few such services. Numerous independent 
living services have been developed since then as federal funding for independent living services 
has increased. A recent review by Donkoh et al. (2006) found that no rigorous evaluations of 
such services have been conducted. Rigorous evaluation of various independent living services is 
needed to inform efforts to improve their effectiveness. 

This report presents findings from a rigorous evaluation of the Early Start to Emancipation 
Preparation (ESTEP)-Tutoring Program in Los Angeles. ESTEP-Tutoring provides up to 50 
hours of remedial one-on-one tutoring in reading and math to foster youths ages 14 to 15 who are 
one to three years behind grade level in either reading or math. The tutoring is provided in the 
youth’s home by tutors who are typically local college students. Other program staff guide the 
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tutors and assess the youths. Another implicit goal of the program is to build an ongoing 
mentoring relationship between the youth and the tutor. We examine its implementation and its 
impact on the youths it serves using a rigorous random- assignment method with a two-year 
follow-up. This is one of four impact reports from a study required by the Foster Care 
Independence Act of 1999, funded by the Children’s Bureau, and directed by the Children’s 
Bureau and the Office of Planning, Research, and Evaluation in the U.S. Department of Health 
and Human Services. 

This work is important for several reasons. First, the work confirms that youth who age out of 
foster care are not doing well and need further attention from the systems that have served them 
before they turned 18 years old. Second, child welfare systems can — and should — rigorously test 
interventions using the best possible evaluation methods. It is possible to conduct rigorous 
evaluation in the child welfare system, and it is crucial to do so if the field is to develop services 
that address the great needs of its children and youth. Finally, it is especially important to do this 
work now. The field has developed a significant number of new services in the past few decades, 
but without rigorous evaluation, it is impossible to know what is truly helping the children and 
fa mi lies in the child welfare system. 

In the remainder of this chapter, we present an overview of the Chafee legislation and the 
evaluation purpose, as well as describe site selection for the evaluation along with research 
questions for the study. We also review the research design and methodology for both the impact 
and process studies. In chapter 2, we describe the ESTEP-Tutoring Program using information 
obtained as part of the process study component of the evaluation. Chapter 3 provides an 
overview of the evaluation’s implementation, including a discussion of service take-up, sample 
development, and a description of the sample. Results of the evaluation’s impact study are 
presented in chapter 4. A discussion of process study findings that shed light on the impact 
findings is also presented in chapter 4. Lastly, chapter 5 provides a discussion of the findings of 
the evaluation and how it relates to the broader field of independent living programs. 



Overview of Legislation and Evaluation Purpose 

The Foster Care Independence Act (FCIA) of 1999 (Public Law 106-169) amended Title IV-E of 
the Social Security Act to create the John Chafee Foster Care Independence Program (CFCIP), 
giving states more funding and greater flexibility in providing support for youth making the 
transition to independent living. The FCIA allocates $140 mi llion per year in independent living 
services funding to states, allows states to use up to 30 percent of these funds for room and 
board, enables states to assist young adults between the ages of 18 and 21 who have left foster 
care, and permits states to extend Medicaid eligibility to former foster children up to age 21. 

State performance is a much higher priority under the FCIA than under earlier iterations of 
federal policy in this area. DHHS is required to develop a set of outcome measures to assess state 
performance in managing independent living programs, and states are required to collect data on 
these outcomes. In addition, the FCIA requires that funding under the statute be set aside for 
evaluations of promising independent living programs: 



3 




The Secretary shall conduct evaluations of such State programs funded under this 
section as the Secretary deems to be innovative or of potential national 
significance. The evaluation of any such program shall include information on the 
effects of the program on education, employment, and personal development. To 
the maximum extent practicable, the evaluations shall be based on rigorous 
scientific standards including random assignment to treatment and control groups. 

The Secretary is encouraged to work directly with State and local governments to 
design methods for conducting the evaluations, directly or by grant, contract, or 
cooperative agreement. (Title IV-E, Section 477 [42 U.S.C. 677], g, 1) 

The language in the FCIA requiring rigorous evaluation of independent living programs reflects 
the acknowledgment by lawmakers that little is known about the effectiveness of these programs. 
In response to this language, the Department of Health and Human Services’ Children’s Bureau 
has contracted with the Urban Institute and its partners — the Chapin Hall Center for Children and 
the National Opinion Research Center — to conduct an evaluation of selected programs funded 
through the CFCIP, the Multi-Site Evaluation of Foster Youth Programs. The goal is to 
determine the effects of independent living programs funded under CFCIP in achieving key 
outcomes for participating youths, including increased educational attainment, higher 
employment rates and stability, better interpersonal and relationship skills, fewer nonmarital 
pregnancies and births, and reduced rates of delinquency and crime. 



Program Site Selection 

In 2001, the Department of Health and Human Services contracted with the Urban Institute and 
its partners — the Chapin Hall Center for Children and the National Opinion Research Center — to 
conduct an evaluability assessment of independent living programs. The goal of this assessment 
was to identify programs that could be rigorously evaluated and to develop evaluation designs 
that would meet the requirements of the authorizing legislation. The evaluation team — in 
coordination with the Children’s Bureau and a federally-appointed technical work group — 
established criteria for selecting sites for the evaluability assessment. The Children’s Bureau 
selected programs to be evaluated. 

To be considered for the evaluation, programs were required to exhibit the following: 

• Programs should take in sufficient numbers of youths to allow for the creation of a 
research sample of adequate size. 

• Programs should have excess demand for services so that randomly assigning youths to a 
control group is possible while serving the same number of youths. 

• Programs should be reasonably stable. 

• Programs should be relatively intensive. 
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Programs should have well-developed theories of intervention (“logic models”), linking 
intended outcomes with intervention activities. 






• Programs should be consistently implemented. 

• Sites should have available data with which to understand the flow of clients and to 
follow clients to assess key outcomes. 

• Relevant decisionmakers should be willing to support participation in a rigorous 
evaluation. 

• Program sites should be willing to make minor changes needed to accommodate the 
research and should be able to maintain them for the full research period. 

The evaluation team conducted this assessment to identify programs suitable for evaluation 
between October 2001 and January 2003 and involved the following: 

• identifying independent living programs in the United States; 

• developing information on critical aspects of these programs; 

• categorizing the programs; 

• selecting programs for further study; 

• visiting the selected programs; 

• applying the criteria for evaluability to selected programs; and 

• recommending programs for evaluation. 

The evaluation team contacted thirty-two states and the District of Columbia and examined 87 
different independent living programs. Site visits were conducted with the 23 programs that 
seemed most promising. Most of the programs did not meet the basic criteria for evaluability; 
that is, most did not have sufficient program intake to allow the creation of a research sample of 
adequate size or the excess demand that makes random assignment possible while serving the 
same number of youths. 

Four independent living programs were selected for inclusion in the evaluation, which used a 
random-assignment design. The selected programs encompass a set of critical independent living 
services and represent a range of program types. The programs include an employment services 
program in Kern County, California, modeled on Temporary Assistance for Needy Families 
work development assistance; an intensive case management and mentoring program in 
Massachusetts; a tutoring and mentoring program; and a classroom-based life skills training 
program, both serving youths in Los Angeles County. Note that these four programs are not 
representative of all of the different types of independent living services available to youth in the 
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United States. Rather, they represent a range of different interventions independent living 
programs use. As a result, the findings from the Multi-Site Evaluation cannot be generalized to 
all independent living programs. It is also important to note that the study team attempted to 
identify a housing program to evaluate and investigated several different housing programs 
located throughout the country. However, low numbers of participants in these programs made 
random assignment difficult and would not provide sufficient samples for the analyses. 

The ESTEP-Tutoring Program provides a service (tutoring) that is offered in numerous locations 
throughout the United States. While ESTEP-Tutoring may provide a typical set of services, 
unusual aspects of the program may provide useful information for other independent living 
programs. At the time of the evaluation, the ESTEP-Tutoring Program was offered in 12 
community colleges throughout Los Angeles County. The program also offers services beyond 
tutoring, including a mentoring relationship with the tutor and access to other independent living 
workshops provided through the ESTEP program. Aside from programmatic aspects, the 
program was selected because of the large number of youths that it serves (between 400 and 500 
youths annually at the time of the evaluability assessment). Finally, ESTEP-Tutoring is an 
oversubscribed service, having nearly twice as many youths referred as program participants. 



Research Questions 

The Multi-Site Evaluation addressed the following research questions: 

• Program impact: What impact does access to the identified intervention have for youth 
compared with similar youths who have access to standard services or “services as usual” 
on key outcomes like self-sufficiency and well-being (e.g., educational skills, 
interpersonal skills, living skills, employment skills, psychosocial well-being)? 

• Program mission: How does the program identify its logic model? Does service 
implementation follow the logic model and mission? 

• Service implementation: How are services implemented? Who performs the service 
delivery? What is the training and experience of staff delivering services? 

• Who is being served: What types of youth are being served? Is there an assessment 
protocol to determine the types and duration of services needed? Who is excluded? Do 
the intended populations receive services? 

• Program challenges: What are barriers to implementation? How can they be ameliorated 
or eliminated? 

• Policy variables: How might external community or state-level variables contribute to 
outcomes achieved by program youth? 
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• Portability of program models: To what extent might these programs be adapted to other 
locales? How transportable are these services and program models to other programs in 
other regions? 



Research Design and Methods 

This evaluation used an experimental design, whereby some youths were randomly assigned to 
have access to the service being evaluated while others have access to “services as usual,” both 
of which vary by site. Youths assigned to the group with access to the service are referred to as 
“treatment group youths” or “ESTEP group youths.” Youths that were assigned to “services as 
usual” are referred to as “control group youths.” A more detailed description of the random- 
assignment process and ESTEP and control groups is presented at the beginning of chapter 3. 

The evaluation consists of two elements: an impact study and a process study. To determine the 
effects of independent living programs on the key outcomes required by the Chafee legislation, 
youths in both the ESTEP and control groups were interviewed in person at three points over the 
course of the evaluation. For the process study, members of the evaluation team visited the sites 
to observe the programs and conduct interviews and focus groups with youths, staff, 
administrators, and service providers. A more in-depth description of the evaluation 
methodology is located in appendix A. 

Impact Study 

The main source of data for identifying program impacts comes from interviews with foster 
youths. For the ESTEP-Tutoring evaluation, we randomly assigned youths who had been 
referred for ESTEP-Tutoring to either treatment, referred to as the “ESTEP group” or control 
group. Our target was to interview 450 youths across ESTEP and control groups at the baseline. 
Each respondent was asked to participate in an initial interview as well as two follow-up 
interviews, with expected first and second follow-up retention rates of 85 and 80 percent, 
respectively. Each follow-up interview was to take place approximately one year after the 
previous interview with that respondent. 

Sample Overview . The analysis sample consists of 445 youths who were referred to the ESTEP- 
Tutoring program. Generally, youths, age 14 to 15, are initially assessed on their reading and 
math skills by an emancipation preparation advisor using The Community College Foundation 
(TCCF)-devised assessments. These assessments produce grade-level equivalent scores. Those 
who are assessed as being one to three years behind their school grade level in either reading or 
math are referred to TCCF for tutoring. 

We far exceeded our response rate expectations. Response rates differ only slightly between the 
ESTEP-Tutoring and control groups. Youths were very cooperative and interested in 
participating, evidenced by the very small number of refusals. Caregiver refusals accounted for 
most of the noninterviews. These were referred to as gatekeeper refusals, because caregivers 
often serve as gatekeepers who provide or limit access to the youths. Forty-six youths (8.7 
percent) were found to be out-of-scope. The vast majority of out-of-scope cases were youths who 
were reunited or living with a legal guardian. 
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More detailed information on response and retention rates and out-of-scope conditions for the 
ESTEP sample population is provided in appendix A. 

Youth Questionnaire. The youth questionnaire is the primary data collection tool used in the 
study. It provides the foundation for the impact study, but also offers critical information about 
youths’ backgrounds and experiences. The evaluation team designed the youth questionnaire 
primarily by using questions from existing surveys. The sources were selected to provide 
questions that had been used extensively and would provide good possibilities to compare with 
other samples. Four surveys provided the bulk of the questions. The Midwest Evaluation of the 
Adult Functioning of Former Foster Youth (the “Midwest Study”) and the National Survey of 
Child and Adolescent Wellbeing (NSCAW) provided questions about child welfare and provide 
comparison samples of foster youths. In particular, the Midwest Study provided a good 
comparison sample of foster youth aging out of care. The National Fongitudinal Survey of 
Youths, 1997 cohort (NFSY97) and the National Fongitudinal Survey of Adolescent Health 
(AddHealth) provided many of the other questions and allowed comparisons with nationally 
representative samples of adolescents aging into their 20s. Special attention to the questionnaire 
design and selection of items was paid so that the core questionnaire could be used with youths 
referred to independent living services at each selected site and so that the same questionnaire 
could be used in each round, with minor variations across rounds. ESTEP and control youths 
were interviewed shortly after referral and random assignment, and follow-up interviews 
occurred one and two years later. 

The questionnaire was designed to take approximately 90 minutes; actual average timings were 
closer to 100 minutes. Most of the interview was conducted with the interviewer asking the 
questions and recording the youths’ responses on a laptop computer. Some sections of the 
questionnaire were administered with audio computer-assisted self-interviewing (ACASI), 
whereby the youth can either read the questions on the computer screen or listen to a recorded 
voice asking the questions. The computer faces the respondent and the interviewer does not see 
the youths’ responses as the youth enters them directly. Sensitive sections of the interview were 
conducted with ACASI. 1 

Where required, the questionnaire was adapted to specific program sites. For example, education 
is a critical part of the ESTEP-Tutoring Program. The questionnaire was tailored to the program 
by including three tests from the Woodcock Johnson Tests of Achievement III: letter- word 
identification (test 1), calculation (test 5), and passage comprehension (test 9). Test 1 (letter- 
word identification) consists of items asking youths to pronounce words and simpler items 
asking youths to identify letters. Test 5 (calculation) is a measure of the youth’s ability to 
perform calculations. The youth completes a workbook with calculation problems of varying 
degrees of difficulty. Test 9 (passage comprehension) consists of passages that the respondent 
reads silently. Each passage has a blank and the youth must complete the sentence. Difficulty 
varies across items on this test, too, with the simplest items consisting of recognizing words and 
following verbal instructions. 



1 The sections administered through ACASI were Substance Abuse, Sexual Behavior, Victimization, and 
Delinquency and Externalizing Behaviors. 
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Outcome Measures. Sections of the youth questionnaire served to identify the services received, 
short- and long-term outcomes, and moderating factors that influence the efficacy of the services 
received. Table 1.1 displays categories of data collection topics (sections of the questionnaire) by 
their purpose for analysis. These topics were primarily addressed in the surveys, though 
qualitative data collected during the process study (described below) also shed light on some 
areas of interest. 

• Population Characteristics. The framework begins with the characteristics of the 
population of interest in each evaluation site, their demographics, and fixed factors, such 
as prior experiences in care and prior victimization. 

• Intervention and Services. The evaluation tests whether an intervention in the site alters 
outcomes of the treatment youths compared with youths receiving the usual or typical 
services. We gathered information on both the focal independent living services (offered 
only to the treatment group) and the other services received by treatment and control 
group youths. 

• Moderating Factors. A set of factors was expected to moderate the effects of the 
interventions. These factors operate at many levels (the youths themselves, the family 
constellation, and the community). These are separated from the characteristics of the 
youths because they may change over time. 

• Short-Term (Intermediate) Outcomes. Early data collection after the intervention will 
establish the short-term outcomes of the treatment and control group youths. These 
outcomes may pick up progress on pathways to the final outcomes of interest (for 
example, education that will ultimately increase success in the labor market) or behaviors 
that affect ultimate outcomes (for example, sexual behaviors that affect fertility and 
health risks). 

• Longer-Term Outcomes. The ultimate goals of the interventions are related to successful 
functioning in adulthood. Key areas mentioned for the evaluation in the Foster Care 
Independence Act include educational attainment, employment, and “personal 
development.” The latter includes physical health, fertility, economic hardship, mental 
health, incarceration, and victimization. 



Table 1.1. Conceptual Framework for Data Collection and Analytic Purposes 




of Questionnaire Sections 




Population 

Characteristics 


Intervention 
and Services 


Moderating 

Factors 


Short-Term 

Outcomes 


Longer-Term 

Outcomes 


Demographics 


Independent 
living services 
of interest 


Relationships 


Employment and 
income 


Employment and 
income 


Prior experiences 
in care 


Other services 


Social support 


Education 


Education 


Prior victimization 




Reading ability 


Health behaviors 


Physical health 
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Table 1.1. Conceptual Framework for Data Collection and Analytic Purposes 
of Questionnaire Sections 



Population 


Intervention 


Moderating 


Short-Term 


Longer-Term 


Characteristics 


and Services 


Factors 


Outcomes 


Outcomes 






Living 

arrangements 


Substance abuse 


Fertility and 
family formation 










Economic 






Substance abuse 


Sexual behavior 


hardship or 

homelessness 






Pro-social and 
other activities 


Delinquency 


Mental health 






Mental health 


Mental health 


Victimization 






Attitudes and 
expectations 










Sense of 
preparedness 







Process Study 

A key component of the evaluation was examining how the programs under evaluation were 
implemented, commonly referred to as a process or implementation analysis. The process study 
played a key role in documenting the nature of the programs, interpreting the findings of the 
impact analysis, and suggesting directions for refining the impact study’s design. Specifically, 
the process analysis describes and analyzes the programs under evaluation by addressing two 
broad areas: the current and changing context and the implementation of the services. Each part 
of the process analysis from the site visits to observational analysis addressed one of these two 
areas. 

Program data have been collected to document the recruitment for and the receipt of services 
under the evaluation. The extent of the program data collected varies by program. However, it 
generally includes data on recruitment (e.g., successful and unsuccessful attempts), service 
participation (e.g., the amount of services that the youths received such as number of classes 
attended), and crossovers (i.e., control group youths who received the service under evaluation). 
The process study also sought to collect case-specific data from public child welfare agency 
workers through a worker survey. The purpose of the worker survey was to collect case-specific 
information about the foster youths, such as their developmental and placement history, the 
services they have received, and caseworkers’ perceptions of their preparedness for 
independence. Low initial response rates resulted in the discontinuation of the caseworker survey 
in Los Angeles County. 

To obtain an in-depth understanding of the programs under evaluation and the broader 
independent living services available to youths in both the control and the ESTEP groups, site 
visits were conducted for each program under evaluation. During the visits, a number of 
semistructured interviews were conducted with administrators from the public child welfare 
agency, private agencies or organizations providing services to youths in the control and ESTEP 
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groups, and other key stakeholders. To gain a full understanding of services and operations from 
all relevant perspectives, semi structured interviews and focus groups were also conducted with 
caseworkers, supervisors, and independent living workers in the public child welfare agency and 
with staff and administrators of the programs under evaluation. Focus groups were conducted 
with youths who had and had not received the services under evaluation. In each site, members 
of the process study team also observed staff working with the programs under evaluation. 

Site visits were conducted in Los Angeles in October 2003 and August 2005. Table 1.2 lays out 
the types and numbers of respondents by qualitative method. Interviews and focus group 
protocols focused on the following areas (although not all topics were appropriate for all 
respondents): program planning, operational aspects, service delivery, and program assessment. 

During the first site visit in October 2003, six members of the evaluation team spent two weeks 
in Los Angeles. During this visit, the team met with administrators, supervisors, caseworkers, 
and independent living coordinators within the Los Angeles County Department of Children and 
Family Services to understand the broader independent living context in the county. The team 
conducted semi structured interviews and focus groups with staff from the ESTEP-Tutoring 
Program operated under contract with Los Angeles County by The Community College 
Foundation to understand the program under evaluation. Focus groups were conducted with 
youths who received ESTEP-Tutoring services. Team members were also able to interview five 
community college program directors, some stakeholders working on emancipation issues in Los 
Angeles County, and several service providers. Finally, the evaluation team observed training for 
ESTEP tutors. 

For the second site visit, four members of the evaluation team spent a week in Los Angeles. The 
purpose of the first visit was to understand the broad context of independent living services and 
the programs under evaluation, and the aim of the second visit was to understand more fully how 
the programs operate. For this reason, the second visit contained more in-depth interviews with 
and observations of key program staff. In addition, the team conducted interviews with key 
Department of Child and Family Services emancipation services staff, transition coordinators, 
and relevant emancipation stakeholders. Focus groups with foster youths were planned; however, 
these groups were poorly attended. After the visit, during September and October 2005, team 
members conducted hour-long individual phone interviews with program staff that conduct 
outreach for the ESTEP-Tutoring Program. 

In preparation for the site visits, TCCF program documents and Department of Child and Family 
Services policies relevant to independent living were collected and reviewed. This document 
review continued throughout the duration of the study in Los Angeles. 



Table 1 . 2 . ESTEP-Tutoring Process Study Respondents 
by Qualitative Method 





Type of Respondent 


First Site Visit 
(October 2003) 


Second Site Visit 
(August 2005) 


Respondents 
by Type (h) 


Individual 


DCFS 




6 


11 


interviews 


administrators/managers 






TCCF program 
administrators/managers 


5 


5 


10 
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Table 1.2. ESTEP-Tutoring Process Study Respondents 
by Qualitative Method 




Type of Respondent 


First Site Visit 
(October 2003) 


Second Site Visit 
(August 2005) 


Respondents 
by Type (n) 


TCCF program staff 


0 


21 


21 


Other stakeholders 


14 


4 


18 


DCFS supervisors 


55 


0 


55 


„ DCFS workers 


54 


8 


62 


Focus groups „„„„ . 

TCCF program staff 


48 


12 


60 


Youths 


5 


4 


9 


Observations TCCF program staff 


3 


6 


9 


Respondents by site visit (n) 


189 


66 


255 


Note: DCFS = Los Angeles County Department of Child and Family Services; TCCF - The Community 
College Foundation 



As discussed earlier, program data were collected to document the recruitment for and receipt of 
services under the evaluation. Data were collected on recruitment into ESTEP-Tutoring (noting 
the reasons a youth did not accept the service) and service participation (including how many 
hours of tutoring youths received). These data were collected for youths assigned to the ESTEP 
and control groups. Examining program data on control group members has allowed the 
evaluation team to identify violations of control group status. These data are presented in chapter 
3 of this report. 



Evaluation Challenges 

To better understand the remainder of this report, it is important to briefly discuss some major 
challenges experienced throughout the evaluation. These challenges will be discussed in greater 
detail in appendix A, but it is necessary to outline some of them here. First, in any evaluation 
with an experimental design, there are inherent difficulties using administrative data to randomly 
assign participants, as well as in maintaining the random assignment (i.e., ensuring that 
participants in the control group do not receive the service or intervention and that participants in 
the ESTEP group do). In addition, imposing upon established procedures in a given program is 
complicated. As discussed at length in chapter 2 and appendix A, the evaluation changed some 
of TCCF’s procedures, which proved challenging. Finally, the evaluation faced significant 
challenges due to characteristics of the foster care population. Similar in some respects to other 
disadvantaged populations, youth in the foster care system are highly mobile both when they are 
in care and once they have been emancipated. These youths also have higher rates of behavioral 
and mental health issues. These are just a few of the many difficulties that make working with 
the foster care population challenging. 
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Report Organization 



This chapter has introduced the purpose and intent of the Multi-Site Evaluation and provided 
overview of the research design of the evaluation, including outcome measures. While the 
impact study is the critical part of the evaluation, it is important first to provide a better 
understanding of the design, context, and structure of the ESTEP-Tutoring program. This 
information is presented in chapter 2. 




Chapter 2. Early Start to Emancipation Preparation (ESTEP)-Tutoring Program: 

Context, Description, and Operations 
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This chapter plays an important role in the Multi-Site Evaluation. Describing the Early Start to 
Emancipation Preparation (ESTEP)-Tutoring program in detail, including program 
implementation, staffing, referral, recruitment, and services provided, offers an understanding of 
how the program operates. This information provides background needed to understand the 
results of the impact study. This chapter begins with an overview of the context within which the 
ESTEP-Tutoring program operates, including state and local demographics, and local policies 
and practices for youths aging out of foster care. The discussion then presents logic models for 
the ESTEP-Tutoring program and the context within which it operates. The chapter turns to a 
detailed description of the ESTEP-Tutoring program, including staffing, referral, recruitment, 
and services provided. Finally, the chapter concludes with a discussion of some challenges to 
service provision. 

In examining the program’s implementation, it is important to keep in mind that social service 
programs often change and adapt to changing contexts as necessary. As a result, some 
descriptions may no longer reflect current operations. The programs may have changed 
operations, reporting structures, or forms used since the research team collected data. However, 
although this report points out some recent changes to the programs’ operations and structures, 
the focus is on the implementation and operation of the programs during the evaluation period. In 
other words, the process study captures the operation of the programs while the youths in the 
study were involved with them. It is important to understand how the programs were being 
implemented for the youths in the evaluation. The research team acknowledges that these 
programs adapt and grow and has noted these developments whenever possible. 



Context for Evaluated Program 

To understand the ESTEP-Tutoring program and the extent to which it has the intended impact 
for foster youth, it is critical to understand the context within which the program operates. The 
ESTEP-Tutoring program operates within the larger context of Los Angeles County, posing a 
number of challenges that might affect outcomes for youths in the study, as well as program 
operations and implementation. The county’s size (both geographically and in population) and its 
diverse population are only two of many factors. In particular, it is important to understand the 
demographics and nature of Los Angeles County, as well as any emancipation services in the 
county. The following section describes demographic characteristics of the foster youth 
population in Los Angeles County and discusses other contextual factors that may affect the 
outcomes of youths in this study. A more in-depth description of the state and local context is 
provided in appendix B . 

State and Local Demographics 

California is the most populous state in the nation, with more than 35 million residents in 2004 
(table B.2). Slightly more than a quarter (27 percent) of the population are under age 18, and 35 
percent are Latino. Just over 80 percent of the population age 25 and older are high school 
graduates, and 10 percent have less than a ninth-grade education. Nineteen percent of children 
and 1 1 percent of families were living below the federal poverty level in 2004, when per capita 
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